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Peer Review at NRSA 




Product Peer Reviews are used to discover defects, as a 
validation technique, and to prepare for formal reviews. 


»L-Peer reviews/inspections are part of the program/project 
management process and system engineering process 

*i Product P4er Reviews can be used on many different® 
products at any phase] in a project life cycle. 


r* Peer reviews are often held prior to formal reviews on completed 
products. 


❖ The results of peer reviews may be addressed at formal document 
reviews. 
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Heasons for 





peer revjews 


• It is difficult for an individual Author or development team 
to spot every mistake or flaw in a complicated piece of 
! work. 

This is not necessarily a reflection on the individuals 
concerned, but because with a new and perhaps eclectic 
subject, an opportunity for improvement may be more 
obvious to someone with special expertise or who simply 
looks affpwith a fresh eye. 


• Showing work to others increases the probability that 
weaknesses will be identified and Improved. 
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NASA uses two different types of peer review: scientific 
peer review (also known as refereeing) and Product Peer 
Refiew. Thi| presentation describes only Product Peer 
Renews. 

The JSC Engineering Directorate Product Peer Review 
process has been used for informajl pilot studies, but has 
nof been baselined yet. 

Similar to the Goddard Space Flight Center’s Engineering Peer 
Review process 
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requirementsior peer 


reviews 


Peer reviews/inspections are part of the program/project 
management process and system engineering process 


NPR 7120.5D, NASA Space Flight Program and Project 
Management Requirements 


NPR 71 23.1 A NASA Systems Engineering Processes and 
Requirements, Appendix G.20 


❖ NASA/SP-2007-6105, NASA Systems Engineering Handbook 
Appendix N 


NPR 7150.2, NASA Software Engineering Requirements 
[SWE-087], [SWE-088], [SWE-089], [SWE-119] 
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uveDemonstration otia Product Peer Review 


The rest of this presentation gives a live demo 


Green slides are presented at the Kick-Off Meeting 


• Red slides are presented at the Product Peer Review 
Panel Meeting 

(Blue slides are part of the NASA PM Challenge 
presentation 

• Volunteers from the audience will form our panel of 
Reviewers 
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Magic Cookie Caper 
Product Peer Review 
Kick-Off Meeting 

Ken Jenks Moderator 
2009 - 02-24 



Agenda 


■ Purpose of today’s meeting 

■ Purpose of this Product Peer Review 

■ Product Peer Review team introductions 

E Expectations for the review meeting and process 

■ Product overview 

■ Procedures and tools used in this peer review 

■ Reviewer homework 

■ Defects 

■ Technical risks 

■ Closing the meeting 


Kick-off Meeting Purposes 


■ The meeting today is a Kick-off Meeting 
intended to 

B Introduce members of the review team 

B Familiarize Reviewers with the product 

Train the Reviewers on the Product Peer 
Review process 
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Purpose of this review 

This Product Peer Review is intended to improve the product 

9 * Compare the product against its requirements, standards 
and specifications, finding defects 

■ Specific suggestions for product improvements 
■ Often used to prepare for formal reviews 

Used as a validation technique 

■ Support the evolving design and development of the 
product 

■ Provide technical insight needed to ensure product and 
process quality 

Provide insight into technical risks 
The product under review: 

Product title, revision level 
Stage in the project life cycle 

The Author will correct all major defects 

■ Minor and trivial defects are addressed as time permits 

At the end of this peer review process, the Moderator will prepare 
and distribute the Product Peer Review Report 



Procedure flow 


Planning 



Panel Meeting 
Announcement 
and Package 



Legend 

Process Stage 
Optional Stage 
Person 

► Stage Transition 

Form 




Master Disposition 
Record 



Based on NASA/SP-2007-6105 
Appendix N 
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Peer Review Team 
Introductions 

■ Moderator 

■ Moderator name, organization, contact info 

■ Author 

Author name, organization, contact info 

■ Reviewers 

Reviewer 1, organization, expertise 
■B Reviewer 2, organization, expertise 

■ Reader 

Reader name, organization 

■ Recorder 

Recorder name, organization 



Expectations 

■ You are expected to do your homework 

■ You are expected to find defects 

HU Any complicated technical product will have defects 
■ Finding zero defects is not acceptable 

■ You are expected to record basic metrics (effort, 
defect lists) 

k You are expected to attend the Product Peer 
Review Panel Meeting 

■ Charge codes for this peer review effort 

I ■ This meeting 1 hour, homework 3 hours, next meeting 2 
hours 

■ Disclosure of vested interests, conflicts of interest 

■ Author’s Line Management stays out of the room 

Helps Reviewers be more objective 
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Expectations: Social 
Aspects 

■ Social dynamics between the Reviewers and the 
Author are a sensitive issue 

■ Reviewers 

■ Not here to show that you’re smarter than the Author 

■ Not here to show that you’re smarter than each other 

■ Thoughtfully select the words you use to raise an issue 

■ Comment about the product and not about the Author 

■ Author 

■ Not here to justify every bit of the work product 
Not here to rationalize away problems 

■ Accept the comments graciously, even if you disagree 

■ We’re all here to improve the product 

■ Nobody walks away mad 



The Reader presents the background 


These charts are usually supplied by the Author 


The product background includes 






❖ Who are the product stakeholders 


❖ How will the product be used 


❖ How does the product fit into its product architecture 


❖ When is the product needed 
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Product Background 


■ “The Magic Cookie Caper” is a movie 
coming out next year. Our company is 
bidding to produce cookies to be sold 
in theaters as a movie tie-in. 

■ This batch of cookies is a prototype. 

Our company gives its pitch to the film 
distribution company next month. 
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• The Reader presentsthese charts 
• =7 ; Wiese CheirtWMfe usually supplied ay the Author 


• Compliance checklists and traceability matrices are 

=\=l 3 el pf0=i - - - I^ =k' J 


•I It’s common fonReviewers to find problems in the 
product’s requirements, standards and specifications 

^ =*^m¥zrep brt Mtfs e defects as product defects 

s Moderator should include a CR form in Data Package 
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Requirements, Standards 
and Specifications 


[MCP-00100] 

There shall be 12 cookies in each batch. 

[MCP-00200] 

At least 5 cookies shall be sugar cookies. 

[MCP-00300] 

At least 5 cookies shall be chocolate chip. 

[MCP-00400] 

At least 4 cookies shall have pink sprinkles. 

[MCP-00500] 

At least 3 cookies shall have blue sprinkles. 

[MCP-00600] 

The cookies shall be round. 

[MCP-00700] 

The cookies shall be between 2 inches and 3 inches in 
diameter. 
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• The Reader presents the product information 


• These charts are usually supplied by the Author 




Product Information 


■ This batch of cookies was prepared by our product 
development team, not a commercial kitchen 

■ We added the Oreos because we found that we can 
get a profitable cross-promotion deal with Nabisco 
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Product Discussion 


■ Reviewers are invited to ask question 
of the Author 

| ■ About the product 
About requirements 
About the standards 
| ■ About the specifications 
About technical risks 


Procedures and tools 


■ EA-WI-038 describes the procedure for this 
Product Peer Review 

Moderator distributes forms to Reviewers 

I ■ During the Product Peer Review Panel Meeting, 
the Recorder will record any additional defects 
discovered 

■ Moderator will use EA Action Item Database 
for action items 

■j ■ Track actions identified in the reviews until they 
are resolved 

■ Moderator will use Microsoft Exchange 
“Shared Tasks” for issue tracking 
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Reviewer Homework 

Inspect the product carefully 

Compare the product against its requirements, 
standards and specifications 

Record major and minor defects that you detect on the 
Individual Defect List 

Use the Trivial Defects Log for typos, grammatical 
changes, spelling errors, formatting problems or other 
trivia 

Take notes about questions you have or risks that 
you’ve identified 

Keep track of the effort spent performing these tasks 

Fill out the Individual Preparation Log 

Send your Individual Defect List and your Individual 
Preparation Log to the Moderator by the due date 

Bring your notes to the Product Peer Review Panel 
Meeting 


Defects 

■ Defect: discrepancy or nonconformity to a 
requirement or specification 

■ Reviewers will identify defects in the product 

Include assumptions made in determining defects 


Severity 

Description 

Major 

An error that would cause a malfunction or prevents attainment of an 
expected or specified result. 

Any error that would in the future result in an approved change request or 
failure report. 

Minor 

A violation of standards, guidelines, or rules that would not result in a 
deviation from requirements if not corrected but could result in minor 
difficulties in terms of operations, maintenance, or future 
development. 

Trivial 

Editorial errors such as spelling, punctuation, and grammar that do not 
cause errors or change requests. 

Recorded as redlines or in the electronic Trivial Defects Log. Presented 
directly to Author at the end of the meeting. 



Technical Risks 


■ Discussions of defects and approaches 
will often reveal technical risks 

■ Author will record technical risks in the 
project risk list 

■ Although a Product Peer Review may reveal 
risks, risk management is not part of the 
Product Peer Review process 


25 


Closing the Meeting 


■ Assignment for Reviewers 

Review the product against its 
requirements 

■ Location of product and requirements 

Individual Preparation Log 
| ■ Trivial Defects Log 
Due date: yyyy/mm/dd 

■ Schedule for the Product Peer Review 
Panel meeting 

yyyy/mm/dd, hh:mm, location 
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And then a 



4 A week between the Kick-Off Meeting and the Product 
Peer Review Panel Meeting gives the Reviewers time to 
do ^rai^sfew ork and turn in their defect lists to the 
| Moderator 

• The Moderator collects the defects together, combining 
d u p I [|a tesf^^^- 

3 The Moderator and the Author can correct some defects 


4 Normally, the Product Peer Review Panel Meeting should 
jp5jf; held ii fgffei same conference room or workstation 
cluster a week later 


k 
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Magic Cookie Caper 
Product Peer Review 
Panel Meeting 


TBD Moderator 

yyyy/mm/dd 
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Agenda 


Purpose of this Product Peer Review 

Product Peer Review team 

Expectations 

Success criteria 

Closing the meeting 



Purpose of this review 


Product Peer Reviews are used 

• to discover defects 

• as a validation technique 

• to prepare for formal reviews 

The product under review: 

• Product title, revision level 

• Stage in the project life cycle 



Peer Review Team 


Moderator 

• Moderator name, organization, contact info 

Author 

• Author name, organization, contact info 

Reviewers 

• Reviewer 1, organization, expertise 

• Reviewer 2, organization, expertise 

Reader 

• Reader name, organization 

Recorder 

• Recorder name, organization 



Expectations 

• This Product Peer Review is intended to improve 
the product 

• Compare the product against its requirements, 
standards and specifications, finding defects 

• Specific suggestions for product improvements 

• Gain insight into the technical risks 

• Record basic metrics (effort, defect lists) 

• Charge codes for this peer review 

• Disclosure of vested interests, conflicts of 
interest 

• Author's Line Management stays out of the room 

• Helps Reviewers be more objective 
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Procedure flow 




Moderator 

Author 

Reviewers 




Reviewers 

Others 


Legend 

Process Stage 
Optional Stage 
Person 

► Stage Transition 




Form 


Based on NASA/SP-2007-6105 
Appendix N 
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Expectations: Social Aspects 

• Social dynamics between the Reviewers and the Author 
are a sensitive issue 

• Reviewers 

• Not here to show that you're smarter than the Author 

• Not here to show that you're smarter than each other 

• Thoughtfully select the words you use to raise an issue 

• Comment about the product and not about the Author 

• Author 

• Not here to justify every bit of the work product 

• Not here to rationalize away problems 

• Accept the comments graciously, even if you disagree 

• We're all here to improve the product 

• Nobody walks away mad 
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• The Reader presents the major defects 


• These charts are usually supplied by the Moderator 



Procedures and tools 


• EA-WI-038 describes the procedure for 
this Product Peer Review 

• Moderator will use EA Action Item 
Database for action items 

• Moderator will use Microsoft Exchange 
"Shared Tasks" for issue tracking 
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Success criteria 

Criteria 

Is the product is complete? 

Does the product conform to the applicable regulations, standards, guidelines, plans, and procedures? 
Have changes to the product been properly implemented so that they affect only the specified areas? 
Is the product is suitable for its intended use? 

Is the product is ready for the next activity? 

Do any defects or discrepancies still exist? 

Are the defects and their recommended resolutions clearly identified? 

Has a list of action items been generated to resolve the defects? 

Do the Product Peer Review Panel members agree to the recommended resolutions of identified 
anomalies? 


Has the meeting been adequately documented? 
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Closing the Meeting 

• Did the product pass its peer review? 

• Determining the need for a Third Hour 

• Schedule for the Product Peer Review Report 
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^ftergh^n eeting 


4 The Author corrects all Major defects 

Author may correct some or all Minor and Trivial defects 

•^The. Moderator prepares the Product Peer Review Report 


j_^From a template 
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•i Contact Ken.Jenks@nasa.gov 
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Requirements 
Magic Cookie Caper 
Batch 3 


[MCP-00100] 

There shall be 12 cookies in each batch. 

[MCP-00200] 

At least 5 cookies shall be sugar cookies. 

[MCP-00300] 

At least 5 cookies shall be chocolate chip. 

[MCP-00400] 

At least 4 cookies shall have pink sprinkles. 

[MCP-00500] 

At least 3 cookies shall have blue sprinkles. 

[MCP-00600] 

The cookies shall be round. 

[MCP-00700] 

The cookies shall be between 2 inches and 3 inches in 
diameter. 




Product Peer Review Panel 
Individual Preparation Log 


Reviewer’s Name: 

Org: 

Phone: 

Email: 

Other: 

Review Product: Magic Cookie Caper - Trial Batch #3 
Date Package Received: 

Date Preparation Completed: 


Hours Spent: 


Date worked on 

Time expended (in Hours) 









Total Hours: 



The moderator needs to receive this form, or a copy, at least prior to the 

scheduled meeting. Please check the appropriate boxes below. 


□ 

Reschedule peer review 

□ 

Do not reschedule peer review 

□ 

The work is not ready for 
peer review 

□ 

1 am prepared for my role in the 
peer review 

□ 

1 need more preparation 
time 

□ 

1 will be prepared in time for the 
peer review 


TPR Panel Individual Preparation Log 20080625 


Page 1 of 1 



Magic Cookie Caper Product Peer Review 
Individual Defect List 

Major or Minor defects only. Trivial defects should be listed on the Trivial Defects Log. 


Comment # 

Severity 

Defect Description 


























Severity Description 

Major An error that would cause a malfunction or prevents attainment of an 
expected or specified result. 

Any error that would in the future result in an approved change request or 

failure report 

Minor A violation of standards, guidelines, or rules that would not result in a 
deviation from requirements if not corrected but could result in minor 
difficulties in terms of operations, maintenance, or future development. 

Trivial Editorial errors such as spelling, punctuation, and grammar that do not cause 
errors or change requests. 

Recorded as redlines or in the electronic Trivial Defects Log. Presented 
directly to Author at the end of the meeting. 





Trivial Defects List - Magic Cookie Caper.xls 
Trivial_lnputs_Log 


<Name of Product> 


Technical Peer Review 
Name of Moderator: <Moderator Name> 


Page 

Section 

Comment 

# 

Starting Line No. 

From: 

Paste original wording here. 

To: 

Enter Requested 
Modification / Addition / Deletion 

Rationale 

Mail 

Code 

Submitter: 
Name / 
Phone 



1 









2 









3 









4 









5 









6 









7 









8 









9 









10 









11 









12 









13 









14 









15 









16 









17 









18 









19 









20 









21 









22 









23 









24 









25 









26 









27 









28 









29 









30 









31 









32 









33 









34 








Date Finalized: 


1 of 2 


Trivial Defects List - Magic Cookie Caper.xls 
Trivial_lnputs_Log 


<Name of Product> 


Technical Peer Review 
Name of Moderator: <Moderator Name> 


Comment 

# 

Starting Line No. 

From: 

Paste original wording here. 

Author's Notes 

Author's 

Disposition 

Author's 

Disposition 

Rationale 

Follow Up? 

Status 

DCC 

Instructions 

1 








2 









3 









4 









5 









6 









7 









8 









9 









10 









11 









12 









13 









14 









15 









16 









17 









18 









19 









20 









21 









22 









23 









24 









25 









26 









27 









28 









29 









30 









31 









32 









33 
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Date Finalized: 
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